Information processing apparatus, information processing method, computer-readable recording medium, and information processing system

ABSTRACT

An information processing apparatus includes a processing unit that generates an operation log of an operation target device operated by a user, by identifying an operation content of the operation target device, based on a captured image obtained by capturing an image of the operation target device, and operation pattern information of a model of the operation target device.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims priority to and incorporates by referencethe entire contents of Japanese Patent Application No. 2021-046548 filedin Japan on Mar. 19, 2021.

FIELD

The present invention relates to an information processing apparatus, aninformation processing method, an information processing program, and aninformation processing system.

BACKGROUND

For example, it has been known that a device such as a flowmeter isprovided in a plant (see Japanese Patent Application Laid-open No.2018-185616).

There is a device that cannot generate or output an operation log. Forexample, such a device cannot save an operation log because of limitedstorage capacity, or cannot transmit an operation log to the outside dueto limited power consumption and the like.

An object of the present invention is to obtain an operation log of adevice, even if the device cannot generate or output an operation log.

SUMMARY

According to one aspect of embodiments, an information processingapparatus comprises a processing unit that generates an operation log ofan operation target device operated by a user, by identifying anoperation content of the operation target device, based on a capturedimage obtained by capturing an image of the operation target device, andoperation pattern information of a model of the operation target device.

According to one aspect of embodiments, an information processing methodcomprises generating an operation log of an operation target deviceoperated by a user, by identifying an operation content of the operationtarget device, based on a captured image obtained by capturing an imageof the operation target device, and operation pattern information of amodel of the operation target device.

According to one aspect of embodiments, a computer-readable recordingmedium stores therein an information processing program that causes acomputer to execute a process of generating an operation log of anoperation target device operated by a user, by identifying an operationcontent of the operation target device, based on a captured imageobtained by capturing an image of the operation target device, andoperation pattern information of a model of the operation target device.

According to one aspect of embodiments, an information processing systemcomprises an image capturing device that captures an image of anoperation target device operated by a user, and an informationprocessing apparatus that generates an operation log of the operationtarget device, by identifying an operation content of the operationtarget device, based on a captured image of the operation target devicecaptured by the image capturing device, and operation patterninformation of a model of the operation target device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a schematic configurationof an information processing system according to an embodiment;

FIG. 2 is a diagram illustrating an example of a schematic configurationof an image capturing device;

FIG. 3 is a diagram illustrating an example of a captured image;

FIG. 4 is a diagram illustrating an example of a schematic configurationof an information processing apparatus;

FIG. 5 is a diagram illustrating an example of model identificationinformation;

FIG. 6 is a diagram illustrating an example of individual identificationinformation;

FIG. 7 is a diagram illustrating an example of operation patterninformation;

FIG. 8 is a diagram illustrating an example of screen transition;

FIG. 9 is a diagram illustrating an example of an operation log;

FIG. 10 is a flowchart illustrating an example of a process performed inthe information processing system;

FIG. 11 is a flowchart illustrating an example of a process performed inthe information processing system;

FIG. 12 is a flowchart illustrating an example of a process performed inthe information processing system;

FIG. 13 is a flowchart illustrating an example of a process performed inthe information processing system;

FIG. 14 is a diagram illustrating an example of a schematicconfiguration of an information processing system according to amodification;

FIG. 15 is a flowchart illustrating an example of a process (informationprocessing method) performed in the information processing system;

FIG. 16 is a diagram illustrating an example of a schematicconfiguration of an information processing system according to amodification;

FIG. 17 is a diagram illustrating an example of a schematicconfiguration of an information processing system according to amodification; and

FIG. 18 is a diagram illustrating an example of a hardware configurationof the information processing apparatus.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment will be described with reference to theaccompanying drawings. The same reference numerals denote the samecomponents, and repeated descriptions will be omitted as appropriate.

FIG. 1 is a diagram illustrating an example of a schematic configurationof an information processing system according to an embodiment. Aninformation processing system 100 generates an operation log of a device1 operated by a user U. There may be various types of the device 1. Forexample, the device 1 may be a flowmeter, a pressure gauge, athermometer, and the like provided in a plant and the like. In thiscase, the user U may be an operator (an on-site worker and the like) ofthe plant. In FIG. 1, the user U is manually operating the device 1while viewing a display screen 1 a of the device 1. The display screen 1a may be a touch panel, and in this case, the user U operates the device1 by touching the display screen 1 a. The device 1 may not be able togenerate, save, or output an operation log, for example.

The information processing system 100 includes an image capturing device2 and an information processing apparatus 3. The image capturing device2 and the information processing apparatus 3 are configured to becommunicable with each other. In this example, the image capturingdevice 2 and the information processing apparatus 3 are connected toeach other via a network N. For example, the network N is the Internet,a wide area network (WAN), a local area network (LAN), and the like.

The image capturing device 2 is an apparatus having an image capturingfunction. For example, the image capturing device 2 is a smartphone, atablet terminal, a laptop, a personal computer (PC), and the likeprovided with a camera and the like. The image capturing device 2 willbe described with reference to FIG. 2.

FIG. 2 is a diagram illustrating an example of a schematic configurationof an image capturing device. The image capturing device 2 includes animage capturing unit 21, a display unit 22, a processing unit 23, astorage unit 24, a communication unit 25, and an operation unit 26.

A captured image of the image capturing unit 21 is referred to as acaptured image 21 a. The captured image 21 a is an image obtained bycapturing an image of the device 1. For example, the captured image 21 ais an image for identifying the model of the device 1, an image foridentifying individual devices 1, an image for identifying the operationcontent of the device 1, and the like. The details will be describedbelow.

In the present disclosure, an “image” and “image capturing” may alsomean “video” and “filming”. In the following explanation, the “image”and “image capturing” may also be read as the “video” and “filming” asappropriate within a range not departing from the gist of the presentdisclosure. The image may be a photograph, a screen shot, a scannedimage, a video frame, or an image created by a content authoringapplication.

FIG. 3 is a diagram illustrating an example of a captured image. Forexample, the captured image 21 a is an image for identifying theoperation content of the device 1 (for example, a frame of video). Thedevice 1 is operated by the user U. The captured image 21 a may alsoinclude information on the image capturing time.

Returning to FIG. 2, the display unit 22 displays various types ofinformation. For example, the display unit 22 displays the capturedimage of the image capturing unit 21 in real time. Moreover, the displayunit 22 also displays information useful for the user operation, ordisplay the generated operation log (which will be described below) ofthe device 1.

The processing unit 23 executes various processes on the generation ofan operation log of the device 1. Unless otherwise specified, the imagecapturing device 2 described in the present disclosure is operated, whenthe components of the image capturing device 2 are controlled (when theentire image capturing device 2 is controlled) by the processing unit23.

The storage unit 24 stores various types of information necessary forthe process performed in the image capturing device 2. For example, theinformation stored in the storage unit 24 includes the captured image 21a and a computer program 24 a. The captured image 21 a is as describedabove. The computer program 24 a is a computer program that causes acomputer to execute a process of the image capturing device 2. Forexample, the application relating to the operation log generation of thedevice 1 is executed by the computer program 24 a.

The communication unit 25 communicates with the information processingapparatus 3. Information can be transmitted and received between theimage capturing device 2 and the information processing apparatus 3. Forexample, the communication unit 25 transmits the captured image 21 a tothe information processing apparatus 3. The captured image 21 a may becaptured by the image capturing unit 21, and transmitted to theinformation processing apparatus 3 in real time. Alternatively, thecaptured image 21 a may be retrieved from the storage unit 24 at anytime, and transmitted to the information processing apparatus 3. Thecommunication unit 25 also transmits the user information describedabove and the like, to the information processing apparatus 3.

The transmission of the captured image 21 a to the informationprocessing apparatus 3 will be further described. Only a part of thecaptured image 21 a of the image capturing unit 21 may be transmitted tothe information processing apparatus 3. For example, in the capturedimage 21 a, the processing unit 23 only extracts an image of a portion(a frame or the like) necessary for identifying the model, identifyingthe individual, and identifying the operation content of the device 1(performs segmentation of moving images or the like). For example, ifthe image is necessary for identifying the operation content, theprocessing unit 23 may only extract an image indicating a change in thedisplay screen 1 a of the corresponding device 1, before and after theoperation. The processing unit 23 may only extract an image including afinger, a touch pen, and the like. The image may be automaticallyextracted using a known image recognition technology and the like, or byuser operation. The communication unit 25 transmits the extracted imageto the information processing apparatus 3. If the amount of change ofthe captured image 21 a in the screen (video screen) is a predeterminedamount or less, the communication unit 25 may not transmit the extractedimage to the information processing apparatus 3. Thus, the communicationload is reduced.

The process such as image extraction performed by the processing unit 23as described above is optional. When all the captured images 21 a of theimage capturing unit 21 are automatically transmitted to the informationprocessing apparatus 3, the process such as image extraction will becomeunnecessary, thereby reducing the processing load of the processing unit23.

The operation unit 26 processes the operation (user operation) of theimage capturing device 2 by the user U. If the display unit 22 describeabove includes a touch panel display and the like, part or all of thefunctions of the operation unit 26 may be incorporated into the displayunit 22.

Returning to FIG. 1, the information processing apparatus 3 generates anoperation log of the device 1, on the basis of the captured image 21 atransmitted from the image capturing device 2. The informationprocessing apparatus 3 will be described with reference to FIG. 4.

FIG. 4 is a diagram illustrating an example of a schematic configurationof an information processing apparatus. The information processingapparatus 3 includes a processing unit 31, a communication unit 32, arecording unit 33, a database 34, and a storage unit 35.

The processing unit 31 executes various processes on the generation ofthe operation log of the device 1. Unless otherwise specified, theinformation processing apparatus 3 described in the present disclosureis operated, when the components of the information processing apparatus3 are controlled (when the entire information processing apparatus 3 iscontrolled) by the processing unit 31. The details will be describedbelow.

The communication unit 32 communicates with the communication unit 25 ofthe image capturing device 2. Information can be transmitted andreceived between the information processing apparatus 3 and the imagecapturing device 2. For example, the communication unit 32 receives thecaptured image 21 a transmitted from the communication unit 25 of theimage capturing device 2, and transmits the operation log generated bythe processing unit 31 to the communication unit 25 of the imagecapturing device 2.

The recording unit 33, the database 34, and the storage unit 35 eachrecord, register (save), and store various types of informationnecessary for the process performed in the information processingapparatus 3, and the like. For example, although not limited, whenviewed from the processing unit 31, the recording unit 33 may be arecording unit for Read. The database 34 may be a database forRead/Write. The storage unit 35 may be a storage unit for Read or astorage unit for Read/Write.

Examples of information to be recorded in the recording unit 33 includemodel identification information 33 a, individual identificationinformation 33 b, and operation pattern information 33 c. The modelidentification information 33 a is information used for identifying themodel of the device 1 from the captured image. For example, the “model”is identified by the type name of the device 1, version information ofthe device 1, product ID, and the like. Even if the type name, versioninformation, product ID, and the like may not be completely the same, aplurality of the devices 1 that are operated and driven in substantiallythe same manner may be treated as the same model.

FIG. 5 is a diagram illustrating an example of model identificationinformation. For example, the model identification information 33 a isinformation in which the “model” and a “model identification image” areassociated (linked) with one another. In this example, the “model” isschematically illustrated as “xxx” and the like. The “modelidentification image” is an image that can identify the model of thedevice 1. In FIG. 5, some appearance images of the device 1 areillustrated as the model identification image.

Returning to FIG. 4, the individual identification information 33 b isinformation used for identifying the individual information of thedevice 1 from the captured image 21 a. The individual information isinformation that can uniquely identify the device 1, and for example, aproduct number (serial number and the like), a tag name, and the like.

FIG. 6 is a diagram illustrating an example of individual identificationinformation. For example, the individual identification information 33 bis information in which the “model”, “individual information”, and an“individual identification image” are associated with one another. The“model” is as described above. In this example, the “individualinformation” is schematically illustrated as “xxx-001” and the like. The“individual identification image” is an image that can identify theindividual information of the device 1. The individual identificationimage may be an image illustrating a unique feature of the device 1. Forexample, the devices 1 of the same model may have outer shapes differentfrom each other, and an image including such a portion is used as theindividual identification image. Examples of the different outer shapesinclude unevenness and the like.

Returning to FIG. 4, the operation pattern information 33 c isinformation for identifying the operation content of the device 1 fromthe captured image 21 a. For example, the operation pattern information33 c is prepared for each model.

FIG. 7 is a diagram illustrating an example of operation patterninformation. For example, the operation pattern information 33 c isinformation in which the “model”, “image-derived information”, and“operation content” are associated with one another. The “model” is asdescribed above. The “image-derived information” is information that canbe obtained from an image of the device of the model. Examples of theimage-derived information include an “operation part” and “display”.

The “operation part” illustrates the operation part of the device.Examples of the operation part include a button to be pressed, an iconto be selected, and the like. In the drawing, the “operation part” isschematically illustrated as a “button A”, an “icon A”, and the like.The button may be a hardware button (physical button) or a softwarebutton.

The “display” illustrates the display of the device. Examples of thedisplay include a character string (including a numeric character andthe like) displayed on a display screen of the device, touch feedback onthe display screen (a change in the button background color of a touchpanel while the touch panel is being touched and the like), screentransition of the display screen, and the like. In the drawing, thedisplay is schematically illustrated as a “character string A”, “touchfeedback A”, a “screen transition A”, and the like. The screentransition will be described with reference to FIG. 8.

FIG. 8 is a diagram illustrating an example of screen transition. Thedisplay screen of the device is changed between various screens such asa menu screen 1 a-1, a monitor screen 1 a-2, and a setting screen 1 a-3.Because the screen is changed by user operation, an operation content isassociated with the screen transition. In this example, pressing a“monitor screen button” is associated with the screen transition fromthe menu screen 1 a-1 to the monitor screen 1 a-2. Pressing a “homebutton” is associated with the screen transition from the monitor screen1 a-2 to the menu screen 1 a-1. Pressing a “setting button” isassociated with the screen transition from the menu screen 1 a-1 to thesetting screen 1 a-3. Pressing a “home button” is associated with thescreen transition from the setting screen 1 a-3 to the menu screen 1a-1. Execution of “determination” or “cancellation” is associated withthe screen transition from the setting screen 1 a-3 to the monitorscreen 1 a-2.

There may be various screens in addition to the menu screen 1 a-1, themonitor screen 1 a-2, and the setting screen 1 a-3 described above.Examples of the other screen include a communication screen and thelike.

Returning to FIG. 7, the “operation content” indicates the operationcontent of the device 1. In this example, the operation content isschematically illustrated as “Y1” and the like. The operation content isassociated with single image-derived information or a combination of aplurality of pieces of image-derived information. For example, a certainoperation content (such as determination) is associated with pressing acertain button (such as a button A). A certain operation content (suchas the execution of a process indicated by the character string A) maybe associated with a combination of displaying a certain characterstring (such as a character string A) and pressing a certain button(such as a button A). For example, the color change of the characterstring “save setting”, which is displayed on the display screen of thedevice, to the color of the touch feedback, is associated with theoperation content of executing the setting. These are mere examples, andvarious types of image-derived information and/or a combination of thevarious types of image-derived information, and various operationcontents may be associated with one another.

In addition to the above, various types of information useful foridentifying the operation content of the device 1 from the capturedimage 21 a are included in the operation pattern information 33 c. Forexample, a pattern of a pointer (a finger, a touch pen, and the like)used for screen operation may also be included in the operation patterninformation 33 c as the image-derived information. The gesture of theuser U and the like may also be included in the operation patterninformation 33 c as the image-derived information. Rotating the rotaryswitch provided in the device 1, dragging on a touch panel, flicking ona touch panel, and the like may also be included in the operationpattern information 33 c as the image-derived information.

Returning to FIG. 4, for example, the model identification information33 a, the individual identification information 33 b, and the operationpattern information 33 c as described above are recorded in therecording unit 33. For example, these pieces of information are obtainedprior to the shipment of the device 1 and the like, and recorded in therecording unit 33.

Examples of the information to be stored in the database 34 include anoperation log 31 a. The operation log 31 a indicates the operationhistory of the device 1. The operation log 31 a is generated by theprocessing unit 31, as will be described below.

FIG. 9 is a diagram illustrating an example of an operation log. Forexample, the operation log 31 a is described by associating “individualinformation”, “time”, and “operation content” with one another. The“individual information” and the “operation content” are as describedabove. The “time” indicates the operation time (date and time, and thelike), and in this example, is schematically illustrated as “t1” and thelike.

In the operation log 31 a, the description of the individual informationis optional. In the operation log 31 a, the individual information maynot be included. Alternatively, the model may be included instead of theindividual information.

Returning to FIG. 4, in the database 34, information (user information)on the user U who uses the information processing system 100 may beregistered. Examples of the user information include user ID, user name,belonging organization, operation location of the device 1, logininformation for using (the application of) the image capturing device 2,a type of application used in the image capturing device, the capturedimage 21 a of the device 1, the image capturing date and time, the imagecapturing location, and the like. For example, by saving such userinformation in the database 34 also, the user information can also bereferred to when the operation log 31 a is reviewed later and the like.

Examples of the information stored in the storage unit 35 include acomputer program 35 a. The computer program 35 a is a computer program(information processing program) that causes a computer to execute aprocess of the information processing apparatus 3. The captured image 21a transmitted from the image capturing device 2 may also be stored inthe storage unit 35.

The processing unit 31 will be described in detail. The processing unit31 generates the operation log 31 a of the device 1, by identifying theoperation content of the device 1, on the basis of the captured image 21a of the device 1, which is an operation target device operated by theuser U, and the operation pattern information 33 c of the model of thedevice 1. As the main process, three processes of the modelidentification, the individual identification, and the operation loggeneration will be sequentially described.

Model Identification

The processing unit 31 identifies the model of the device 1. In anembodiment, the processing unit 31 identifies the model of the device 1,on the basis of the captured image 21 a and the model identificationinformation 33 a recorded in the recording unit 33.

In this example, the captured image 21 a may be an image captured toidentify the model of the device 1. The processing unit 23 of the imagecapturing device 2 generates information that prompts the user U tocapture such an image (for example, an appearance image of the device1), and causes the display unit 22 to display the image. By useroperation, the image capturing unit 21 of the image capturing device 2captures an image of the device 1, and the communication unit 25transmits the captured image 21 a to the information processingapparatus 3.

For example, the processing unit 31 retrieves a model identificationimage that matches with the captured image 21 a transmitted from theimage capturing device 2 as described above, from the modelidentification information 33 a. In the image retrieval, various knownmethods such as image pattern matching may be used. The processing unit31 identifies the model associated with the model identification imagethat matches with the captured image 21 a, as the model of the device 1.

In the image retrieval described above, various algorithms for imagerecognition and the like may be used. A learned model may also be usedfor image recognition, and this will be described below when theoperation log generation is discussed.

The model may also be identified without using the captured image 21 a.For example, model information indicating the model may be transmittedto the information processing apparatus 3 from the image capturingdevice 2, and the model indicated in the model information may beidentified as the model of the device 1, which is an operation logtarget. The model information may also be input to the image capturingdevice 2 by user operation.

In addition to the above, various methods capable of identifying themodel may be used, and this will be described below when the individualidentification is discussed.

The communication unit 32 may also transmit the information on the modelidentified by the processing unit 31 to the communication unit 25 of theimage capturing device 2. The display unit 22 of the image capturingdevice 2 may display the information on the model. The user U canconfirm whether the model identification is appropriately performed, andoperate the image capturing device 2 (operate application and the like)to proceed to the next process, or if necessary, perform the modelidentification again.

Individual Identification

The processing unit 31 identifies individual information of the device1, which is an operation log generation target. The individualinformation identification may only be performed on the device 1 thatbecomes the operation log generation target for the first time. When theindividual information is once identified, for example, the informationin which the device 1 and the individual information are associated withone another, is registered in the database 34 (initial individualregistration is completed), and thereafter, the operation log can beobtained by linking the operation log with the registered individualinformation.

In an embodiment, the processing unit 31 identifies the individualinformation of the device 1, on the basis of the captured image 21 a andthe individual identification information 33 b recorded in the recordingunit 33.

In this example, the captured image 21 a may be an image captured toidentify the individual information of the device 1. The processing unit23 of the image capturing device 2 may generate information that promptsthe user U to capture such an image (for example, an image illustratinga unique feature of the device 1), and cause the display unit 22 todisplay the information. By user operation, the image capturing unit 21of the image capturing device 2 captures an image of the device 1, andthe communication unit 25 transmits the captured image 21 a to theinformation processing apparatus 3.

For example, the processing unit 31 compares between the individualidentification image of the model identified in advance, and thecaptured image 21 a transmitted from the image capturing device 2 asdescribed above. In image comparison, various known methods such asimage pattern matching may be used. The processing unit 31 identifiesthe individual information associated with the individual identificationimage that matches with the captured image 21 a, as the individualinformation of the device 1, which is an operation log generationtarget. As described above, the individual identification image is animage illustrating a unique feature of the device 1 (such asunevenness), and an individual identification method based on such afeature may also be referred to as object fingerprint authentication andthe like (for example, Japanese Patent Application No. 2017-543479).

In addition to the image comparison as described above, various methodscapable of identifying the individual information may be used. Forexample, an image recognition process may be performed on the capturedimage 21 a. Various algorithms for image recognition and the like may beused. A learned model may also be used for image recognition, and thiswill be described below when the operation log generation is discussed.

The image recognition may include character recognition. The characterrecognition may include recognition of a character string and the likeused for identifying individual information. By recognizing such acharacter string and the like, the individual information is identified.Such a character string and the like may be extracted from a name plateof the device 1; a seal, a label, or the like affixed to the device 1;an image of a display screen such as the display screen 1 a of thedevice 1; and the like. The individual information may also beidentified by reading a barcode, a quick response (QR) code (registeredtrademark), or the like provided on the device 1.

In character recognition, various algorithms such as optical characterrecognition (OCR) may be used. A plurality of algorithms correspondingto different character display formats (various formats such as 7 seg,11 seg, 14 seg, and 16 seg may be included in addition to dot matrix)may be prepared. For example, according to the identified model, theprocessing unit 31 may extract a character using an optimal algorithm.

The method using the image recognition described above may also beapplied to the model identification described above.

The individual information may also be identified without using thecaptured image 21 a. For example, the individual information may betransmitted to the information processing apparatus 3 from the imagecapturing device 2, and identified as the individual information of thedevice 1, which is an operation log target. The individual informationmay also be input to the image capturing device 2 by user operation.

Moreover, the individual information may be identified from acombination of the identified model and the positional information ofthe device 1 (for example, positional information of the image capturingdevice 2 near the device 1). The positional information is provided byglobal positioning system (GPS) information, terrestrial magnetisminformation, or a combination of the GPS information and the terrestrialmagnetism information. For example, the positional information isobtained by the image capturing device 2, and transmitted to theinformation processing apparatus 3. Because making use of suchpositional information has been known (for example, Japanese PatentApplication Laid-open No. 2018-31599), the detailed description will beomitted here.

The communication unit 32 may also transmit the individual informationidentified by the processing unit 31 to the communication unit 25 of theimage capturing device 2. The display unit 22 of the image capturingdevice 2 may display the individual information. The user U can confirmwhether the individual identification is appropriately performed, andoperate the image capturing device 2 (operate application and the like)to proceed to the next process, or if necessary, perform the individualidentification again.

The model identification and the individual identification describedabove may be performed in parallel (simultaneously). For example, if theQR code or the barcode provides both of the model and individualinformation, the model and individual information can be identifiedsimultaneously by reading the QR code or the barcode only once. It is tobe noted that the individual identification is optional. Depending onthe usage (utilization purpose and the like) of the operation log, theindividual identification is necessary or unnecessary. For example, ifthe user wishes to distinguish between different mounting locations orthe like even if the devices 1 of the same model are used, theindividual identification may be performed.

Operation Log Generation

The processing unit 31 generates the operation log 31 a of the device 1,by identifying the operation content of the device 1, on the basis ofthe captured image 21 a and the model identification information 33 a ofthe identified model.

In this example, the captured image 21 a is an image captured toidentify the operation of the device 1. More specifically, the capturedimage 21 a is an image of the device 1 operated by the user U (FIG. 3and the like). The processing unit 23 of the image capturing device 2may generate information that prompts the user U to capture such animage, and cause the display unit 22 to display the information. By useroperation, the image capturing unit 21 of the image capturing device 2captures an image of the device 1, and the communication unit 25transmits the captured image 21 a to the information processingapparatus 3. As described above, the processing unit 23 may only extractan image of a portion (a frame and the like) necessary for identifyingthe operation content, and the image may be transmitted by thecommunication unit 25.

The processing unit 31 performs an image recognition process on thecaptured image 21 a, and identifies the operation content of the device1, on the basis of an image recognition result of the captured image 21a, and the operation pattern information 33 c. As the image recognitionprocess, an image recognition process capable of obtaining theimage-derived information described above with reference to FIG. 7 isused. Examples of the image recognition result include the image-derivedinformation (operation part and display of the device 1 and the like).More specifically, the image recognition result may include pressing abutton, selecting an icon, displaying a character string, touchfeedback, rotating a rotary switch, dragging on a touch panel, flickingon a touch panel, screen transition, and the like on the device 1.

For example, by the image recognition process, the character stringdisplayed on the display screen 1 a of the device 1 is extracted, and ascreen type of the display screen 1 a (setting screen and the like), theposition of a finger of the user, a touch pen, and the like areidentified. The processing unit 31 retrieves the operation patterninformation 33 c using the extracted result, the identified result, orthe like as a retrieval key, and identifies the operation content. Asdescribed above with reference to FIG. 7, the operation content isassociated with each of the extracted result, the identified result, andthe like, or a combination of the extracted result, the identifiedresult, and the like. Thus, the operation content is identified when theprocessing unit 31 refers to the operation pattern information 33 c.Execution of cancellation that does not affect the operation of thedevice 1 may also be identified.

The processing unit 31 may also identify the operation content of thedevice 1, on the basis of a combination of the screen transition withthe other result in the image recognition result, and the operationpattern information 33 c. For example, the processing unit 31 mayidentify the operation content of the device 1, on the basis of acombination of at least one of pressing a button, selecting an icon,displaying a character string, touch feedback, rotating a rotary switch,dragging on a touch panel, and flicking on a touch panel on the device1, with the screen transition, and the operation pattern information.Consequently, it is possible to identify the operation content, even itis difficult to uniquely identify the operation content only through thescreen transition and the like.

A specific example will be described with reference to FIG. 8 again.When the device 1 is activated, the monitor screen 1 a-2 that displaysmeasurement values is displayed. When the display screen 1 a is changedto display the menu screen 1 a-1, on the basis of the image recognitionresult, it is identified that the display screen 1 a is changed from themonitor screen 1 a-2 to the menu screen 1 a-1. Pressing a menu buttonthat causes the screen transition is identified as the operationcontent. On the other hand, there are two operation contents ofdetermination and cancellation that cause the setting screen 1 a-3 tochange to the monitor screen 1 a-2. In this manner, if a plurality ofoperation contents are associated with a single screen transition, it ispossible to identify the operation content by combining the screentransition with the other result in the image recognition result asdescribed above, instead of only through the screen transition. Forexample, it is possible to identify the operation content as thedetermination operation, by combining with the identification such aspressing a button corresponding to determination.

Returning to FIG. 4, various known algorithms and the like may be usedfor image recognition. In an embodiment, a learned model may be used.For example, the learned model is generated (machine learned) usingtraining data such that when the captured image 21 a is input, the imagerecognition result described above is output. The processing unit 31inputs the data corresponding to the image capturing unit 21 to thelearned model, and obtains the image recognition result from the learnedmodel.

The training data may include an image for learning. The image forlearning may be classified into a type of the device 1, a type of screendisplayed on the display screen 1 a of the device 1 (setting screen,monitor screen, communication screen, operation screen, and the like), aselection state of the displayed content (touch feedback), an alarmissuance state, and the like. Images for learning with differentconditions such as the presence of wiring, the presence of an operatingfinger, and the like may also be prepared.

In learning the positional identification of the finger, to be able tocope with different ethnic groups, images for learning including variousfingers of different colors, shapes, and the like may be prepared, andincluded in the training data. To be able to cope with gloved fingers,touch pens of various shapes, and the like, images for learningincluding the gloved fingers, touch pens of various shapes, and the likemay be prepared, and included in the training data. Images for learningincluding a finger with a mark and the like may also be prepared, andincluded in the training data.

The learned model as described above may also be used for the imagerecognition in the model identification and the individualidentification described above.

The processing unit 31 generates the operation log 31 a using theidentified operation content. An example of the operation log 31 a is asdescribed above with reference to FIG. 9. For example, the processingunit 31 generates the operation log 31 a described by associating theidentified individual information, time, and the identified operationcontent with one another. For example, the time information is obtainedfrom the captured image 21 a transmitted from the image capturing device2. As described above, the description of the individual information isoptional. For example, the processing unit 31 may also generate theoperation log 31 a by associating the model, time, and the operationcontent with one another, instead of using the individual information.

The processing unit 31 registers (saves) the generated operation log 31a in the database 34. Because the image capturing device 2 cancommunicate with the information processing apparatus 3, for example,the user U can use the operation log 31 a registered in the database 34of the information processing apparatus 3, by operating the imagecapturing device 2.

The processing unit 31 may register all the generated operation log 31 ain the database 34, or may only register a part of the generatedoperation log 31 a in the database 34. For example, in the generatedoperation log 31 a, only a portion on the specific function, operation,or the like of the device 1 may be registered in the database 34. Anobject to be registered may be set as appropriate.

The processing unit 31 may also perform a statistical process on thevarious generated operation logs 31 a or on the various operation logs31 a registered in the database 34. For example, frequency of thespecific operation content or the like may be calculated. Examples ofthe specific operation content include an operation content that maylower the operation efficiency such as a cancel operation and a redooperation. For example, the specific operation content may be used forimproving the operation design of the device 1.

For example, in this manner, the operation log is generated. Thegenerated operation log may be fed back to the user U. For example, theoperation log 31 a generated by the processing unit 31 is transmitted tothe image capturing device 2 in real time, or the operation log 31 aregistered in the database 34 is transmitted to the image capturingdevice 2 at any time. The display unit 22 of the image capturing device2 displays the operation log 31 a. The processing unit 23 may generatedisplay information in various formats including the operation log 31 a,and cause the display unit 22 to display the information. The user U canuse the displayed operation log 31 a for confirmation such as preventionof debugging and erroneous operation. The processing unit 23 may alsofilter the displayed operation log 31 a. For example, filtering contentis set by the user U.

FIG. 10 to FIG. 13 are flowcharts each illustrating an example of aprocess (information processing method) performed in the informationprocessing system.

In FIG. 10, as the main processes of the information processing system100, the model identification, the individual identification, and theoperation log generation described above are illustrated. At step S1,the model is identified. At step S2, the individual information isidentified. At step S3, the operation log is generated. The steps willbe described with reference to FIG. 11 to FIG. 13. The descriptions ofthe same contents as those described above will be omitted asappropriate.

FIG. 11 illustrates a detailed flow of step S1 (FIG. 10). The step S1includes step S11 to step S16.

At step S11, the image capturing device 2 transmits a captured image foridentifying the model to the information processing apparatus 3. Theinformation processing apparatus 3 receives the captured imagetransmitted from the image capturing device 2.

At step S12, the information processing apparatus 3 retrieves the modelidentification image that matches with the received captured image. Theprocessing unit 31 of the information processing apparatus 3 retrievesthe model identification image that matches with the captured image 21 atransmitted from the image capturing device 2, from the modelidentification image included in the model identification information 33a in the recording unit 33. As described above, the model may beidentified by the image retrieval.

At step S13, the process is branched according to whether the model isidentified by the image retrieval. If the model is identified by theimage retrieval (Yes at step S13), the process (process at step S1) inthe flowchart is finished. If the model is not identified by the imageretrieval (No at step S13), the process proceeds to step S14. The modelmay not be identified when the image is unclear or other reasons, forexample.

At step S14, the information processing apparatus 3 transmits aretransmission request to the image capturing device 2. The processingunit 31 of the information processing apparatus 3 generatesretransmission request information (such as a command), and thecommunication unit 32 transmits the retransmission request to thecommunication unit 25 of the image capturing device 2. The communicationunit 25 of the image capturing device 2 receives the retransmissionrequest, and the display unit 22 displays information indicating thatthe retransmission request is received and the like. The user U operatesthe image capturing device 2, and captures an image for identifying themodel again or inputs the model information.

At step S15, the image capturing device 2 transmits the captured imageor the model information to the information processing apparatus 3. Theinformation processing apparatus 3 receives the captured image or themodel information.

At step S16, the process is branched according to whether the imageretrieval is necessary. If the captured image is retransmitted by theimage capturing device 2, the image retrieval is necessary (Yes at stepS16), and the process moves to step S12 again. If the model informationis transmitted by the image capturing device 2, the model is identified,and the image retrieval is unnecessary (No at step S16). Hence, theprocess (process at step S1) in the flowchart is finished.

For example, in this manner, the model is identified. In the flowdescribed above, the model information may also be transmitted at stepS11. In this case, the process on the image retrieval at step S12 andthe like will be skipped as appropriate.

FIG. 12 illustrates a detailed flow of step S2 (FIG. 10). The step S2includes step S21 to step S27.

At step S21, the information processing apparatus 3 obtains theindividual identification image from the recording unit. The processingunit 31 of the information processing apparatus 3 obtains the individualidentification image of the model identified at step S1 described above,from the individual identification information 33 b of the recordingunit 33.

At step S22, the image capturing device 2 transmits the captured imagefor identifying the individual, to the information processing apparatus3. The information processing apparatus 3 receives the captured imagetransmitted from the image capturing device 2.

At step S23, the information processing apparatus 3 compares images. Theprocessing unit 31 of the information processing apparatus 3 comparesbetween the individual identification image obtained at step S21described above, and the captured image transmitted from the imagecapturing device 2 at step S22 described above. As described above, theindividual information may be identified by the image comparison.

At step S24, the process is branched according to whether the individualinformation is identified by the image comparison. If the individualinformation is identified by the image comparison (Yes at step S24), theprocess (process at step S2) in the flowchart is finished. If theindividual information is not identified by the image comparison (No atstep S24), the process proceeds to step S25. For example, the individualinformation may not be identified when the image is unclear or otherreasons.

At step S25, the information processing apparatus 3 transmits aretransmission request to the image capturing device 2. The user Uoperates the image capturing device 2, and captures an image foridentifying the individual information again or inputs the individualinformation.

At step S26, the image capturing device 2 transmits the captured imageor the individual information to the information processing apparatus 3.The information processing apparatus 3 receives the captured image orthe individual information.

At step S27, the process is branched according to whether the imagecomparison is necessary. If the captured image is retransmitted by theimage capturing device 2, the image comparison is necessary (Yes at stepS27). Hence, the process moves to step S23 again. If the individualinformation is transmitted by the image capturing device 2, theindividual information is identified, and the image comparison isunnecessary (No at step S27). Hence, the process (process at step S2) inthe flowchart is finished.

For example, in this manner, the individual information is identified.In the flow described above, the individual information may also betransmitted at step S22. In this case, the process on the imagecomparison at step S23 and the like will be skipped as appropriate.

FIG. 13 illustrates a detailed flow of step S3 (FIG. 10). The step S3includes step S31 to step S35 and the like.

At step S31, the information processing apparatus 3 obtains operationpattern information. The processing unit 31 of the informationprocessing apparatus 3 obtains the operation pattern information 33 c ofthe model identified at step S1 described above, from the recording unit33.

At step S32, the image capturing device 2 prepares a captured image foridentifying the operation content. The image capturing unit 21 of theimage capturing device 2 captures an image of the device 1 operated bythe user U. According to the needs, the processing unit 23 only extractsa part of the captured image 21 a of the image capturing unit 21.

At step S33, the image capturing device 2 transmits the captured imageto the image processing apparatus 3. The communication unit 25 of theimage capturing device 2 transmits the captured image 21 a prepared atstep S32 described above, to the communication unit 32 of the imageprocessing apparatus 3. The communication unit 32 of the informationprocessing apparatus 3 receives the captured image 21 a.

At step S34, the information processing apparatus 3 identifies theoperation content. The processing unit 31 of the information processingapparatus 3 identifies the operation content of the device 1, on thebasis of the operation pattern information 33 c obtained at step S31described above, and the captured image 21 a transmitted from the imagecapturing device 2 at step S33 described above.

At step S35, the information processing apparatus 3 generates anoperation log, and registers the generated operation log in thedatabase. The processing unit 31 of the information processing apparatus3 generates the operation log 31 a on the basis of the operation contentidentified at step S34 described above, and registers the operation log31 a in the database 34.

The processes from step S32 to step S35 described above are repeatedlyexecuted. For example, if the processes from step S32 to step S35 areexecuted for the second time, the processes are illustrated as stepS32-2 to step S35-2. If the process on the operation log generation isfinished when the captured image is no longer transmitted from the imagecapturing device 2 and the like, the process (process at step S3) in theflowchart is finished.

For example, in this manner, the operation content of the device 1 isidentified, and the operation log 31 a of the device 1 is generated.

According to the information processing system 100 describe above, it ispossible to generate and obtain (collect) the operation log 31 a of thedevice 1, even if the device 1 cannot generate or output an operationlog. There is no need to add a firmware function and the like to thedevice 1. Because the model identification, the individualidentification, the operation log generation, and the like areautomatically performed using the captured image 21 a obtained bycapturing an image of the device 1, it is possible to reduce the timeand effort of the user U to minimum. Even if identification information(such as a barcode and a QR code) is not marked on the main body or thelike, it is possible to identify the individual information and includethe individual information in the operation log 31 a. The information(captured image 21 a) of the device 1 can be obtained without beingconnected to or without communicating with the device 1, and is linkedto the external information (such as information in the informationprocessing apparatus 3). Hence, the processing load of the device 1 doesnot increase. Because the operation log 31 a is registered (saved) inthe database 34, for example, it is possible to confirm the operation(includes setting change and the like) performed by the user U on thedevice 1 in the past.

As described above, an embodiment of the present disclosure has beendescribed. However, the disclosed technology is not limited to theembodiment described above. Some modifications will now be described.

In an embodiment, as an evidence of the operation log 31 a, the capturedimage 21 a (video) that indicates the whole operation of the device 1may be saved. To save the captured image 21 a (video), a dedicateddatabase may be used. This will be described with reference to FIG. 14.

FIG. 14 is a diagram illustrating an example of a schematicconfiguration of an information processing system according to amodification. For example, an information processing system 100A isdifferent from the information processing system 100 (FIG. 1) inincluding an image capturing device 2A instead of the image capturingdevice 2, and in further including a storage database 4.

The storage database 4 is connected to the image capturing device 2 andthe information processing apparatus 3 via the network N. During theoperation log generation, the image capturing device 2A also transmitsthe captured image of the image capturing unit 21 to the storagedatabase 4. The storage database 4 saves the captured image. Thefunction of the storage database 4 may also be incorporated into theinformation processing apparatus 3 (for example, into the database 34).

In the information processing system 100A, in the process of theoperation log generation (step S3 in FIG. 10 and FIG. 13), a process ofsaving the captured image 21 a in the storage database 4 is added.

FIG. 15 is a flowchart illustrating an example of a process (informationprocessing method) performed in the information processing system. Forexample, the process at step S3A is different from that at step S3 (FIG.13) in further including processes from step S41 to step S44.

At step S41, the image capturing device 2A starts transmitting acaptured image. The image capturing unit 21 of the image capturingdevice 2A obtains the captured image 21 a of the device 1 operated bythe user U, and transmits the captured image 21 a to the storagedatabase 4. The storage database 4 receives the captured image 21 a.

At step S42, the storage database 4 starts saving the captured image.The storage database 4 saves the captured image 21 a transmitted fromthe image capturing device 2A.

The details of the processes from step S31 to step S35 and the like areas described above with reference to FIG. 12. Hence, the descriptionwill not be repeated here.

At step S43, the image capturing device 2A finishes transmitting thecaptured image 21 a to the storage database 4.

At step S44, the storage database 4 finishes saving the captured image21 a.

For example, in this manner, the captured image 21 a (video) thatindicates the whole operation of the device 1 performed by the user U issaved in the storage database 4 as an evidence of the operation log 31a.

In the above, the captured image 21 a is transmitted to and saved in thestorage database 4 in real time. However, the captured image 21 a mayalso be stored in the storage unit 24 of the image capturing device 2,and then collectively transmitted to and saved in the storage database 4afterward (for example, at step S43).

The storage database 4 may also save the captured image 21 a by adding atime stamp of the operation log generation time (operation logacquisition time) to the captured image 21 a. Consequently, it ispossible to retrieve the captured image 21 a, which will be an evidenceof the operation log 31 a at any optional time, and the like.

In an embodiment, the functions of the image capturing device 2 and theinformation processing apparatus 3 may be integrated into one of theimage capturing device 2 and the information processing apparatus 3. Forexample, the function of the information processing apparatus 3 may beincorporated into the image capturing device 2. Alternatively, thefunction of the image capturing device 2 may be incorporated into theinformation processing apparatus 3. By doing so, a network environmentfor connecting the image capturing device 2 and the informationprocessing apparatus 3 will become unnecessary. This will be describedwith reference to FIG. 16.

FIG. 16 is a diagram illustrating an example of a schematicconfiguration of an information processing system according to amodification. For example, in an information processing system 100B, thefunctions of the image capturing device 2 and the information processingapparatus 3 that have been described above are integrated into an imagecapturing device 2B (or in an information processing apparatus 3B).Hence, it is possible to generate the operation log of the device 1,even if there is no communication via the network N as illustrated inFIG. 1 described above.

In an embodiment, the communication function with the informationprocessing apparatus 3 at the image capturing device 2 side may beprovided outside of the image capturing device 2. This will be describedwith reference to FIG. 17.

FIG. 17 is a diagram illustrating an example of a schematicconfiguration of an information processing system according to amodification. For example, in an information processing system 100C, animage capturing device 2 c is connected to a communication apparatus 5.The communication apparatus 5 is communicably connected to theinformation processing apparatus 3 via the network N. The imagecapturing device 2C communicates with the information processingapparatus 3 using the communication apparatus 5. For example, a camera,a video camera, and the like may be used as the image capturing device2C. A PC and the like may be used as the communication apparatus 5.Similar to what has been described above, it is possible to obtain theoperation log 31 a of the device 1, by combining an apparatus having animage capturing function and an apparatus having a communicationfunction.

In an embodiment, the process performed by the information processingapparatus 3 may be implemented by cloud computing and the like. Forexample, part or all of the functions of the processing unit 31, therecording unit 33, and the database 34 of the information processingapparatus 3 may be provided on an external apparatus of the informationprocessing apparatus 3. The information processing apparatus 3 mayexecute the above-described process, by communicating with such anexternal apparatus. The above-described process may also be executed,when the image capturing device 2 directly communicates with such anexternal apparatus.

FIG. 18 is a diagram illustrating an example of a hardware configurationof the information processing apparatus. For example, a computer or thelike having such a hardware configuration functions as the informationprocessing apparatus 3 described above. Examples of the hardwareconfiguration include a communication apparatus 3 a, a display apparatus3 b, a storage apparatus 3 c, memory 3 d, a processor 3 e, and the likeconnected to each other via a bus or the like. For example, the storageapparatus 3 c is a hard disk drive (HDD), a read only memory (ROM), arandom access memory (RAM), and the like. The memory 3 d may also beincluded in the storage apparatus 3 c.

The communication apparatus 3 a is a network interface card and thelike, and enables communication with other apparatuses. For example, thedisplay apparatus 3 b is a touch panel, a display, and the like. Thestorage apparatus 3 c functions as the recording unit 33, the database34, the storage unit 35, and the like.

The processor 3 e causes a computer to function as the informationprocessing apparatus 3, by reading the computer program 35 a (FIG. 4)from the storage apparatus 3 c and the like, and developing the computerprogram 35 a on the memory 3 d. The function includes the function ofthe processing unit 31 as described above.

The computer program 35 a can be distributed via a network such as theInternet. Moreover, the computer program 35 a may be recorded on acomputer-readable recording medium such as a hard disk, a flexible disc(FD), a compact disc read-only memory (CD-ROM), a magneto-optical disk(MO), a digital versatile disc (DVD), and the like, and may be executedby being read out from the recording medium by a computer.

In addition to the information processing apparatus 3, for example, anapparatus such as the image capturing device 2 may also have the samehardware configuration as described above.

For example, the technology described above can be identified asfollows: as described with reference to FIG. 1 to FIG. 9 and the like,the information processing apparatus 3 includes the processing unit 31that generates the operation log 31 a of the operation target device 1operated by the user U, by identifying the operation content of theoperation target device 1, on the basis of the captured image 21 aobtained by capturing an image of the operation target device 1, and theoperation pattern information 33 c of the model of the operation targetdevice 1.

According to the information processing apparatus 3 described above, theoperation log 31 a of the operation target device 1 is generated fromthe captured image 21 a of the operation target device 1. Thus, it ispossible to obtain the operation log 31 a of the device 1 (the device 1that does not have an operation log sweep function, and the like), evenif the device 1 cannot generate or output an operation log.

As described with reference to FIG. 7 and the like, the operationpattern information 33 c is information in which the model, theimage-derived information that can be obtained from the image of thedevice of the model, and the operation content are associated with oneanother. The processing unit 31 may identify the operation content ofthe operation target device 1, on the basis of the image recognitionresult of the operation log 31 a and the operation pattern information33 c. For example, the processing unit 31 can identify the operationcontent of the operation target device 1, by using the image recognitionresult and the operation pattern information 33 c in this manner.

As described with reference to FIG. 7 and the like, the image-derivedinformation in the operation pattern information 33 c may include atleast one of the operation part and display of the device, and the imagerecognition result may include at least one of the operation part of theoperation target device 1, and display of the operation target device 1.The image-derived information may include at least one of pressing abutton, selecting an icon, displaying a character string, touchfeedback, rotating a rotary switch, dragging on a touch panel, flickingon a touch panel, and screen transition on a device. The imagerecognition result may include at least one of pressing a button,selecting an icon, displaying a character string, touch feedback,rotating a rotary switch, dragging on a touch panel, flicking on a touchpanel, and screen transition on the operation target device 1. Forexample, it is possible to identify the operation content of theoperation target device 1, by using such image-derived information aloneor in combination with other pieces of information.

As described with reference to FIG. 7, FIG. 8, and the like, theprocessing unit 31 may identify the operation content of the operationtarget device 1, on the basis of a combination of at least one ofpressing a button, selecting an icon, displaying a character string,touch feedback, rotating a rotary switch, dragging on a touch panel, andflicking on a touch panel with the screen transition on the operationtarget device 1 in the image recognition result, and the operationpattern information 33 c. Consequently, it is possible to identify theoperation content of the operation target device 1, even if it isdifficult to uniquely identify the operation content only through thescreen transition and the like.

As described with reference to FIG. 5 and the like, the processing unit31 may identify the model of the operation target device 1, on the basisof the captured image 21 a and the model identification image (forexample, included in the model identification information 33 a of therecording unit 33), and identify the operation content of the operationtarget device 1, on the basis of the captured image 21 a and theoperation pattern information 33 c of the identified model.Consequently, it is possible to identify the model of the operationtarget device 1, from the captured image 21 a.

As described with reference to FIG. 6, FIG. 9, and the like, theprocessing unit 31 may identify the individual information of theoperation target device 1, on the basis of the captured image 21 a andthe individual identification image (for example, included in theindividual identification information 33 b of the recording unit 33)illustrating a unique feature of the device, and generate the operationlog 31 a by associating the identified individual information with theidentified operation content. Consequently, it is possible to identifythe individual information of the operation target device 1 from thecaptured image 21 a, and generate the operation log 31 a identified bythe individual information.

As described with reference to FIG. 4 and the like, the processing unit31 may register the generated operation log 31 a in the database 34 sothat the user U can use the operation log 31 a. Consequently, it ispossible to feed back the generated operation log 31 a to the user U.

The information processing method described with reference to FIG. 10 toFIG. 13 and the like, is also an embodiment. The information processingmethod generates the operation log 31 a of the operation target device 1operated by the user U, by identifying the operation content of theoperation target device 1, on the basis of the captured image 21 aobtained by capturing an image of the operation target device 1, and theoperation pattern information 33 c of the model of the operation targetdevice (step S3, step S34, and step S35). As described above, by usingsuch an information processing method, it is also possible to obtain theoperation log 31 a of the device 1, even if the device 1 cannot generateor output an operation log.

The computer program 35 a described with reference to FIG. 4, FIG. 18,and the like is also an embodiment. The computer program 35 a is aninformation processing program that causes a computer to execute aprocess of generating the operation log 31 a of the operation targetdevice 1 operated by the user U, by identifying the operation content ofthe operation target device 1, on the basis of the captured image 21 aobtained by capturing an image of the operation target device 1, and theoperation pattern information 33 c of the model of the operation targetdevice 1. As described above, by using such a computer program 35 a, itis also possible to obtain the operation log 31 a of the device 1, evenif the device 1 cannot generate or output an operation log. As describedwith reference to FIG. 18 and the like, the recording medium recordedwith the computer program 35 a is also an embodiment.

The information processing system 100 described with reference to FIG. 1and the like is also an embodiment. The information processing system100 includes the image capturing device 2 that captures an image of theoperation target device 1 operated by the user U, and the informationprocessing apparatus 3 that generates the operation log 31 a of theoperation target device 1, by identifying the operation content of theoperation target device 1, on the basis of the captured image 21 a ofthe operation target device 1 captured by the image capturing device 2,and the operation pattern information 33 c of the model of the operationtarget device 1. As described above, by using such an informationprocessing system 100, it is also possible to obtain the operation log31 a of the device 1, even if the device 1 cannot generate or output anoperation log.

The disclosed technology may be identified as follows: the informationprocessing system 100 includes the image capturing device 2 thatcaptures an image of the device 1, a related program (for example, thecomputer program 35 a) that causes a computer to receive the capturedimage 21 a captured by the image capturing device 2 (for example, viaapplication of the image capturing device 2), the recording unit 33 inwhich the model (product ID and the like) and the display screeninformation and the like linked to the model are recorded, and thedatabase for registering the operation log 31 a of the device 1. Therelated program identifies the model by analyzing the received capturedimage 21 a, retrieves one or more of information on the displaycharacter string, display icon, information on the image, information onthe type of display screen, and information on the screen transitionlinked to the model, from the recording unit 33, and obtains theoperation log 31 a of the device 1 using the retrieved information.

The image capturing unit 21 of the image capturing device 2 may capturean image including at least one of the QR code affixed to the device 1,the display screen 1 a, all or part of the device 1, shape, displayedcharacter string, barcode, name plate, color, icon, displayed image, atype of screen being displayed, gesture information of the user U, andthe like, as the captured image 21 a.

The captured image 21 a may be obtained by means of a photograph, ascreen shot, a scanned image, a video frame, an image or a video (movingimages) created by a content authoring application, and the like.

The image capturing device may have a configuration in which aphotographing device such as a video and a camera is combined with a PCand the like.

All or part of the functions of the recording unit 33, the database 34,and the computer program 35 a may be provided in the image capturingdevice 2, in the information processing apparatus 3 connected to theimage capturing device 2 via the network N, in the storage database 4,in another external storage, and the like.

In the image analysis of the captured image 21 a (for example, the imagerecognition process performed by the processing unit 31 of theinformation processing apparatus 3, and the like), one or more of theshape of the device 1, the displayed character string, identificationinformation such as a QR code, color, icon, displayed image, a type ofscreen being displayed, and the like, may be analyzed, and used as aretrieval key of the recording unit 33.

In the identification of the model (product ID and the like), forexample, the model information manually input to the image capturingdevice 2 by the user U may be obtained by communication in addition toimage analysis. In this case, there is no need to capture an image foridentifying the model.

In the recording unit 33, information (for example, the modelidentification information 33 a) including at least one of a displaycharacter string, a type of display screen, a type of display icon, atype of input component such as a hardware/software button, a type ofmenu, a display method of touch feedback, and screen transition on thedevice may be recorded.

Image data of an identification unit for individual identification,feature point information (unevenness and the like) extracted from theimage data, and the like may be recorded in or associated with therecording unit 33.

At least one of the user ID of the user, user name, belongingorganization, operation location, login information for using theapplication in the image capturing device, the type of application beingused, a captured image of the operation target device 1, the imagecapturing date and time, and the image capturing location may beregistered in the database 34.

The captured image 21 a; the model identified from the captured image 21a; the individual information; information on the captured image 21 a,the model identified from the captured image 21 a, and the individualinformation input to the application (image capturing device 2); theimage analysis result of the captured image 21 a; the generatedoperation log 31 a, and the like may be registered (collected and thelike) in the database such as the database 34 and the storage database4. The information registered in the database and the like may beretrieved.

The information registered in the database, the retrieved information,and the like may be statistically processed.

For example, the recording unit 33 may be a storage area that can onlyRead in which information is recorded in advance. The database 34 may bea storage area that can Read and Write such that information can becollected, retrieved, and the like.

What is claimed is:
 1. An information processing apparatus comprising: aprocessing unit that generates an operation log of an operation targetdevice operated by a user, by identifying an operation content of theoperation target device, based on a captured image obtained by capturingan image of the operation target device, and operation patterninformation of a model of the operation target device.
 2. Theinformation processing apparatus according to claim 1, wherein theoperation pattern information is information in which a model,image-derived information capable of being obtained from an image of adevice of the model, and an operation content are associated with oneanother, and the processing unit identifies the operation content of theoperation target device, based on an image recognition result of thecaptured image and the operation pattern information.
 3. The informationprocessing apparatus according to claim 2, wherein the image-derivedinformation in the operation pattern information includes at least oneof an operation part of a device and display of a device, and the imagerecognition result includes at least one of an operation part of theoperation target device and display of the operation target device. 4.The information processing apparatus according to claim 2, wherein theimage-derived information includes at least one of pressing a button,selecting an icon, displaying a character string, touch feedback,rotating a rotary switch, dragging on a touch panel, flicking on a touchpanel, and screen transition on a device, and the image recognitionresult includes at least one of pressing a button, selecting an icon,displaying a character string, touch feedback, rotating a rotary switch,dragging on a touch panel, flicking on a touch panel, and screentransition on the operation target device.
 5. The information processingapparatus according to claim 4, wherein the image recognition resultincludes at least one of pressing the button, selecting the icon,displaying the character string, the touch feedback, rotating the rotaryswitch, dragging on the touch panel, and flicking on the touch panel onthe operation target device, and the screen transition on the operationtarget device, and the processing unit identifies the operation contentof the operation target device, based on a combination of at least oneof pressing the button, selecting the icon, displaying the characterstring, the touch feedback, rotating the rotary switch, dragging on thetouch panel, and flicking on the touch panel on the operation targetdevice, with the screen transition on the operation target device, inthe image recognition result, and the operation pattern information. 6.The information processing apparatus according to claim 1, wherein theprocessing unit identifies the model of the operation target device,based on the captured image and a model identification image, andidentifies the operation content of the operation target device, basedon the captured image and operation pattern information of theidentified model.
 7. The information processing apparatus according toclaim 1, wherein the processing unit identifies individual informationof the operation target device, based on the captured image and anindividual identification image illustrating a unique feature of thedevice, and generates the operation log by associating the identifiedindividual information with the identified operation content.
 8. Theinformation processing apparatus according to claim 1, wherein theprocessing unit registers the generated operation log in a database sothat the user is capable of using the operation log.
 9. An informationprocessing method comprising: generating an operation log of anoperation target device operated by a user, by identifying an operationcontent of the operation target device, based on a captured imageobtained by capturing an image of the operation target device, andoperation pattern information of a model of the operation target device.10. A computer-readable recording medium having stored therein aninformation processing program that causes a computer to execute aprocess of generating an operation log of an operation target deviceoperated by a user, by identifying an operation content of the operationtarget device, based on a captured image obtained by capturing an imageof the operation target device, and operation pattern information of amodel of the operation target device.
 11. An information processingsystem comprising: an image capturing device that captures an image ofan operation target device operated by a user, and an informationprocessing apparatus that generates an operation log of the operationtarget device, by identifying an operation content of the operationtarget device, based on a captured image of the operation target devicecaptured by the image capturing device, and operation patterninformation of a model of the operation target device.